Interactive input systems that allow users to inject input (i.e. digital ink, mouse events etc.) into an application program using an active pointer (eg. a pointer that emits light, sound or other signal), a passive pointer (eg. a finger, cylinder or other suitable object) or other suitable input device such as for example, a mouse or trackball, are known. These interactive input systems include but are not limited to: touch systems comprising touch panels employing analog resistive or machine vision technology to register pointer input such as those disclosed in U.S. Pat. Nos. 5,448,263; 6,141,000; 6,337,681; 6,747,636; 6,803,906; 7,232,986; 7,236,162; and 7,274,356 assigned to SMART Technologies ULC of Calgary, Alberta, Canada, assignee of the subject application, the contents of which are incorporated by reference; touch systems comprising touch panels employing electromagnetic, capacitive, acoustic or other technologies to register pointer input; tablet personal computers (PCs); laptop PCs; personal digital assistants (PDAs); and other similar devices.
Multi-touch interactive input systems that receive and process input from multiple pointers using machine vision are also known. One such type of multi-touch interactive input system exploits the well-known optical phenomenon of frustrated total internal reflection (FTIR). According to the general principles of FTIR, the total internal reflection (TIR) of light traveling through an optical waveguide is frustrated when an object such as a pointer touches the waveguide surface, due to a change in the index of refraction of the waveguide, causing some light to escape from the touch point. In a multi-touch interactive input system, a machine vision system captures images including the point(s) of escaped light, and processes the images to identify the position of the pointers on the waveguide surface based on the point(s) of escaped light for use as input to application programs. One example of an FTIR multi-touch interactive input system is disclosed in United States Patent Application Publication No. 2008/0029691 to Han.
In multiple or single-touch interactive input systems, graphic objects, such as the background or “canvas”, and “widgets” overlying the canvas including windows, icons, menus, pictures, text, lines, curves or shapes are displayed on the display surface. Depending upon the application, there may be a number of graphic widgets displayed at different x-y (width-height) positions on the two-dimensional canvas, one or more of which may appear to overlap another in the limited sense that windows in a computer graphical user interface might be made to appear to overlap.
In some applications, the background is a three-dimensional environment having, in addition to dimensions in the x-y plane, a non-zero depth or a “thickness” in the z-direction. In such an environment, three-dimensional graphic widgets, such as for example figurines of animals, vehicles and/or plants, may be manipulated by rotation, translation, scaling, and so forth. Such three-dimensional environments are considered distinct from known two-dimensional environments that give merely the appearance of stacking of graphic widgets in the z-direction by their occlusion of other graphic widgets. For example, in such a two-dimensional environment, a graphic widget has a zero-thickness and occupies only the zero-thickness x-y plane at whichever z-position it has been placed. In contrast, in a three-dimensional environment, a graphic widget has a nonzero-thickness and accordingly occupies/passes through a range of x-y planes. In a three-dimensional environment, the z-position of a graphic widget represents the z-position of a particular point on the graphic widget, such as for example its centre of gravity.
As would be understood, in the physical world an object reacts to a person's actions depending upon its physical properties and the forces applied to it. However, in the virtual world, how objects represented as graphic widgets react to human intervention depends on a particular mapping of human movement to computer feedback. Since interactive input systems such as tabletop displays with a touch panel upon which the environment and the graphic widgets are displayed afford direct touches for interaction, such interaction can be made to seem more physical for a user. Such directness of interaction provides great potential for taking advantage of the physical and virtual aspects of the system.
For example, a graphic widget may be selected by contacting the touch surface with a pointer such as a finger at a location exactly corresponding to the location at which the graphic widget is displayed. Alternatively, as described in U.S. patent application Ser. No. 12/240,919 to Edward Tse, filed on Sep. 29, 2008 entitled “METHOD FOR SELECTING AND MANIPULATING A GRAPHICAL OBJECT IN AN INTERACTIVE INPUT SYSTEM, AND INTERACTIVE INPUT SYSTEM EXECUTING THE METHOD” and assigned to SMART Technologies ULC of Calgary, Alberta, the assignee of the subject application, the content of which is incorporated herein by reference, a graphic widget may be selected by tracking one or more pointer touch points associated with the background and in the event that one or more touch points coincide with a location of the graphic widget, associating the coinciding one or more touch points with the graphic widget thereby to select the graphic widget.
With the graphic widget having been selected, while the pointer remains in contact with the touch surface the touch point remains associated with the graphic widget as though the pointer were sticky or “stuck” to the graphic widget, and can thereby be used to manipulate the graphic widget. For example, the user can manipulate the selected graphic widget by moving the pointer across the display surface thereby to translate the selected graphic widget. Manipulation based on tracking of one or more pointers may be processed using a force-based metaphor. For example, in the two-dimensional environment, a Rotation and Translation (RNT) algorithm using the metaphor of an opposing force may be used to move the graphic widget in the x-y plane, and rotate the graphic widget about the z axis based on tracking of a single touch point and treating the single touch point as though it were “stuck” to the graphic widget while the touch point remains in contact with the touch surface. Details of an RNT algorithm are set out in “Fluid Orientation on a Tabletop Display: Integrating Rotation and Translation” by Russell Kruger, et al, and published in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 2005, Portland, Oreg.), ACM Press, pages 601-610, 2005, the content of which is incorporated herein by reference in its entirety.
In a three-dimensional environment, it is often desirable to manipulate three-dimensional graphic widgets in three dimensions. For example, it would be useful to be able to rotate a graphic widget about the x, y, and z axes, and translate the graphic widget in the x, y and z directions. Prior techniques addressing this functionality, such as that shown by Mark Hancock, Sheelagh Carpendale and Andy Cockburn (2007) in “Shallow-depth 3D interaction: design and evaluation of one-, two and three-touch techniques.” In CHI '07: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1147-1156. ACM, New York, N.Y., USA. ISBN 978-1-59593-593-9, the contents of which are incorporated herein by reference in their entirety, are capable only of five degrees of movement: three for rotation and two for translation.
Some prior art techniques have resorted to tracking execution by a user of predefined touch gestures in order to manipulate objects. Other techniques involve selection from menus. While these techniques achieve some improvements, they suffer respective disadvantages: a user must break the strong physical manipulation metaphor in order to make a gesture or select from a menu, and in the case of gestures must learn or be taught those gestures that are supported by the system.
As will be appreciated, although the above-described techniques improve the user experience of manipulating graphic widgets, the possibilities of user interaction with interactive input systems have not been fully exploited. It is therefore an object to provide a novel method for manipulating a graphic widget in a three-dimensional environment displayed on a touch panel of an interactive input system, and a novel interactive input system embodying the method that provide improvements over the prior art.